AgentKube-Kamaji
Introduction
AgentKube-Kamaji is a specialized agent within the Fulcrum Core ecosystem designed to manage Kubernetes tenant clusters through Kamaji. It serves as a bridge between Fulcrum Core's centralized management system and Kamaji's multi-tenant Kubernetes architecture, enabling seamless provisioning and lifecycle management of Kubernetes clusters.
The system enables organizations to:
- Deploy and manage multiple tenant Kubernetes clusters through a single control plane
- Automate the provisioning of Kubernetes resources across different environments
- Efficiently utilize infrastructure by sharing control plane components
- Monitor and collect metrics from tenant clusters
- Integrate with Proxmox for VM-based tenant cluster nodes
- Apply standardized configurations and security policies across tenant clusters
- Process jobs from Fulcrum Core's job queue system
Context
AgentKube-Kamaji operates within the broader Fulcrum ecosystem, serving as a specialized agent that connects Fulcrum Core to Kamaji for Kubernetes cluster management. The following diagram illustrates how AgentKube-Kamaji interacts with other components in the system:
graph TB
FC((Fulcrum Core API))
AK[AgentKube-Kamaji]
KP[Kamaji Parent Cluster]
PX[Proxmox]
subgraph Tenant Clusters
TC1[Tenant Cluster 1]
TC2[Tenant Cluster 2]
TC3[Tenant Cluster 3]
end
FC -->|Deploy & Control| AK
AK -->|Report Status & Metrics| FC
AK -->|Manage TCPs| KP
AK -->|Provision VMs| PX
KP -->|Create & Manage| TC1
KP -->|Create & Manage| TC2
KP -->|Create & Manage| TC3
Components and Their Roles
AgentKube-Kamaji
AgentKube-Kamaji is a Fulcrum agent specialized in managing Kubernetes clusters through Kamaji. It:
- Authenticates with Fulcrum Core and processes job requests
- Interacts with Kamaji API to create and manage Tenant Control Planes (TCPs)
- Provisions and manages VMs on Proxmox for tenant cluster nodes
- Collects and reports metrics from Kamaji and tenant clusters
- Handles the lifecycle of Kubernetes tenant clusters
- Processes service operations (create, update, delete) for Kubernetes resources
- Maintains secure communications with Fulcrum Core through token-based authentication
- Reports service status and operational data back to Fulcrum Core
Kamaji Parent Cluster
The Kamaji Parent Cluster is a Kubernetes cluster that hosts the control planes for tenant clusters. It:
- Provides the infrastructure for running tenant control planes
- Manages the etcd instances for each tenant cluster
- Efficiently shares resources across multiple tenant control planes
- Handles networking between tenant control planes and worker nodes
- Utilizes a LoadBalancer (MetalLB) to expose tenant control planes
Tenant Control Planes (TCPs)
Tenant Control Planes represent individual Kubernetes clusters managed by Kamaji. They:
- Contain the Kubernetes control plane components (API server, controller manager, scheduler)
- Expose an API endpoint for worker nodes to connect to
- Maintain isolation between different tenant clusters
- Allow custom configuration of Kubernetes versions and settings
- Support addons like CoreDNS, kube-proxy, and konnectivity
Model
This section outlines the key entities in the AgentKube-Kamaji system and their relationships.
Class Diagram
classDiagram
FulcrumAgent <|-- AgentKube : extends
AgentKube --> KamajiClient : uses
AgentKube --> ProxmoxClient : uses
AgentKube --> JobHandler : uses
AgentKube --> MetricsReporter : uses
JobHandler --> KamajiClient : uses
JobHandler --> ProxmoxClient : uses
MetricsReporter --> KamajiClient : uses
class FulcrumAgent {
id : UUID
name : string
state : enum[New|Connected|Disconnected|Error|Disabled]
tokenHash : string
heartbeat()
reportMetrics()
pollJobs()
}
class AgentKube {
vmManager : VMManager
kamajiClient : KamajiClient
proxmoxClient : ProxmoxClient
updateVMResources()
getVMStateCounts()
getJobStats()
}
class KamajiClient {
createTCP(name, spec)
deleteTCP(name)
updateTCP(name, spec)
getTCP(name)
listTCPs()
getTCPStatus(name)
}
class ProxmoxClient {
createVM(spec)
deleteVM(id)
startVM(id)
stopVM(id)
getVMStatus(id)
getVMMetrics(id)
}
class JobHandler {
processCreateJob(serviceId, spec)
processDeleteJob(serviceId)
processStartJob(serviceId)
processStopJob(serviceId)
processUpdateJob(serviceId, spec)
pollAndProcessJobs()
}
class MetricsReporter {
collectClusterMetrics()
collectNodeMetrics()
collectPodMetrics()
report()
}
class VMManager {
createVM(name, spec)
deleteVM(id)
startVM(id)
stopVM(id)
updateVMResources()
getStateCounts()
}
class TenantControlPlane {
name : string
namespace : string
version : string
replicas : int
serviceType : string
port : int
addons : map[string]object
status : TCPStatus
}
class TCPStatus {
phase : enum[Pending|Running|Failed]
controlPlaneEndpoint : string
conditions : Condition[]
observedGeneration : int
}
Entities
Core Components
-
AgentKube
- Extension of the base Fulcrum Agent specialized for Kubernetes management
- Implements the Fulcrum agent interface for integration with Fulcrum Core
- Contains specialized components for interacting with Kamaji and Proxmox
- Manages the lifecycle of tenant Kubernetes clusters
- Processes jobs from Fulcrum Core and translates them to Kamaji/Proxmox operations
-
KamajiClient
- Handles interactions with the Kamaji API
- Creates, updates, and deletes Tenant Control Planes (TCPs)
- Monitors TCP status and health
- Configures TCP specifications like Kubernetes version, replicas, and addons
-
ProxmoxClient
- Manages interactions with Proxmox VE API
- Provisions and manages virtual machines for Kubernetes nodes
- Monitors VM status and resource utilization
- Handles VM lifecycle operations (create, start, stop, delete)
-
JobHandler
- Processes jobs from Fulcrum Core job queue
- Translates Fulcrum service operations to Kamaji/Proxmox operations
- Manages the job lifecycle from claiming to completion
- Reports job status back to Fulcrum Core
-
MetricsReporter
- Collects metrics from Kamaji parent cluster and tenant clusters
- Gathers VM performance data from Proxmox
- Reports resource utilization to Fulcrum Core
- Monitors health of control planes and worker nodes
-
VMManager
- Manages the lifecycle of VMs used for Kubernetes nodes
- Tracks VM states and resource utilization
- Coordinates with Proxmox for VM operations
- Updates VM resources based on service requirements
Kamaji Resources
-
TenantControlPlane (TCP)
- Represents a tenant Kubernetes cluster configuration
- Defines control plane specifications like replicas and Kubernetes version
- Configures networking, addons, and other cluster settings
- Maps to Kubernetes custom resources in the Kamaji parent cluster
-
TCPStatus
- Tracks the current state of a tenant control plane
- Provides endpoint information for connecting to the tenant cluster
- Contains conditions that reflect the health of the TCP
- Used for monitoring and error detection
Architecture
AgentKube-Kamaji is built with Go, leveraging its concurrency model and performance characteristics to efficiently manage Kubernetes clusters. The system integrates with both Fulcrum Core and Kamaji to provide comprehensive Kubernetes management.
Architectural Layers
AgentKube-Kamaji follows a modular architecture organized into the following layers:
-
Agent Core Layer
- Implements the Fulcrum agent interface
- Handles authentication and communication with Fulcrum Core
- Manages agent lifecycle and state
- Coordinates operations between different components
-
Job Processing Layer
- Polls and processes jobs from Fulcrum Core
- Translates job actions to Kamaji and Proxmox operations
- Manages the job lifecycle (claim, process, complete/fail)
- Reports job status back to Fulcrum Core
-
Kamaji Integration Layer
- Interacts with Kamaji API to manage tenant clusters
- Creates and configures Tenant Control Planes
- Monitors TCP status and health
- Processes Kubernetes-specific operations
-
VM Management Layer
- Interfaces with Proxmox API to manage virtual machines
- Provisions and configures VMs for Kubernetes nodes
- Monitors VM status and resource utilization
- Handles VM lifecycle operations
-
Metrics Collection Layer
- Gathers metrics from Kamaji, tenant clusters, and Proxmox
- Processes and normalizes collected data
- Reports metrics to Fulcrum Core
- Monitors system health and performance
Deployment Architecture
AgentKube-Kamaji is typically deployed alongside a Kamaji parent cluster, with access to both Fulcrum Core and Proxmox. The deployment architecture follows this pattern:
-
Kamaji Parent Cluster
- High-availability Kubernetes cluster (typically 3+ nodes)
- Hosts tenant control planes as deployments
- Uses etcd for storing cluster state
- Contains MetalLB for LoadBalancer services
- Runs cert-manager for certificate management
-
AgentKube-Kamaji
- Runs as a standalone service with access to the Kamaji API
- Connects to Fulcrum Core for job processing and reporting
- Interfaces with Proxmox for VM management
- Maintains persistent connections for efficient operation
-
Proxmox Cluster
- Provides virtualization infrastructure for Kubernetes nodes
- Hosts VMs for tenant cluster worker nodes
- Manages compute, storage, and network resources
- Provides API for VM lifecycle management
Workflow Diagrams
Tenant Cluster Creation Workflow
sequenceDiagram
participant FC as Fulcrum Core
participant AK as AgentKube-Kamaji
participant KC as Kamaji API
participant PX as Proxmox API
FC->>AK: Create Kubernetes Cluster Job
AK->>AK: Process Job Parameters
AK->>KC: Create Tenant Control Plane
KC-->>AK: TCP Created
AK->>PX: Provision Worker Node VMs
PX-->>AK: VMs Created
AK->>KC: Get TCP Connection Info
KC-->>AK: TCP Endpoint and Credentials
AK->>PX: Configure VMs as Kubernetes Nodes
PX-->>AK: VMs Configured
AK->>KC: Verify Cluster Health
KC-->>AK: Cluster Status
AK->>FC: Report Job Completion
Note over AK,FC: Include TCP Endpoint and Credentials
Tenant Cluster Monitoring
sequenceDiagram
participant AK as AgentKube-Kamaji
participant KC as Kamaji API
participant PX as Proxmox API
participant FC as Fulcrum Core
loop Every Metric Interval
AK->>KC: Get TCP Status
KC-->>AK: TCP Status Data
AK->>KC: Get Cluster Metrics
KC-->>AK: Cluster Metrics Data
AK->>PX: Get VM Resource Usage
PX-->>AK: VM Resource Data
AK->>AK: Process and Normalize Metrics
AK->>FC: Report Metrics
FC-->>AK: Acknowledge
end
Implementation Details
Agent Integration with Fulcrum Core
AgentKube-Kamaji integrates with Fulcrum Core using the standard agent interface, which includes:
- Token-based authentication
- Status updates through heartbeat mechanism
- Job polling and processing
- Metrics reporting
- Service lifecycle management
Kamaji Management
The agent manages Kamaji Tenant Control Planes through the Kubernetes API, using:
- Custom Resource Definitions (CRDs) for TCP management
- Kubernetes API server for resource creation and updates
- Watch mechanisms for monitoring TCP status changes
- Resource management for control plane components
Proxmox VM Management
For managing virtual machines on Proxmox, the agent:
- Uses the Proxmox VE API for VM operations
- Manages VM templates for quick node provisioning
- Monitors VM resource utilization
- Handles VM lifecycle events
Configuration and Customization
AgentKube-Kamaji supports various configuration options:
- TCP template customization
- Node resource allocation
- Kubernetes version selection
- Network configuration options
- Custom addons and extensions
Conclusion
AgentKube-Kamaji extends Fulcrum Core's capabilities to manage Kubernetes clusters using Kamaji's multi-tenant architecture. By leveraging both Kamaji for Kubernetes management and Proxmox for infrastructure provisioning, it provides a comprehensive solution for deploying and managing Kubernetes at scale.
The integration with Fulcrum Core enables consistent management across different service types, while the specialized Kubernetes and VM management capabilities ensure efficient operation of tenant clusters. This combination makes AgentKube-Kamaji a powerful tool for organizations that need to provision and manage multiple Kubernetes environments through a unified interface.